Composite reception/emission apparatus

ABSTRACT

A composite reception and emission apparatus comprises: a plurality of first-type devices that receives or emits energy; a plurality of second-type devices whose type is different from a type of the first-type devices; and a base body. The first-type devices and the second-type devices measure a full solid angle, and are arranged on the base body such that a device closest to each of the first-type devices in an orientation space is at least one of the second-type devices, and a device closest to each of the second-type devices in the orientation space is at least one of the first-type devices.

TECHNICAL FIELD

The present invention relates to a composite reception and emissionapparatus, and more particularly, to a technique for arranging devicesof a plurality of types on a single base body.

BACKGROUND ART

In recent years, a portable information terminal apparatus such as acell phone and a smartphone has been miniaturized while being providedwith various additional functions for differentiation among them.Especially, a camera function has been enhanced so that a wide range ofimages and videos can be captured by a wide-angle lens mounted thereon.For example, Patent Literature 1 discloses, “The surrounding camera isconfigured by mounting each one camera on each face of a polyhedron suchas a regular dodecahedron and a picked-up image over the entirecircumference can be obtained by each camera. By sequentially connectingthe images picked up by the adjacent cameras, one full circumferentialimage can be obtained. However, the surrounding cameras cannot beassembled in a way that projection centers of the cameras can completelybe in matching each other. Then in the case of jointing the picked-upimages, adjusting dynamically the jointed positions of the adjacentpicked-up images depending on far and near of the object can eliminatemissing and conspicuous joints around the borders between the images soas to generate a seamless surrounding scenery” (excerpted fromAbstract).

CITATION LIST Patent Literature

Patent Literature 1: JP-A-2001-204015

SUMMARY OF INVENTION Technical Problem

In the ranging method according to Patent Literature 1, a distance to asubject is assumed, the jointed portions between the captured images aredynamically adjusted, and a distance when the jointed portions aresmoothly connected is determined as an estimated distance to thesubject. Since the ranging method according to Patent Literature 1 is anindirect method as described above, the accuracy of the obtaineddistance is not sufficient. Accordingly, in addition to the imagingfunction, it is desirable to separately provide a ranging function fordirectly measuring the distance to the subject. Since a frame used inthe surrounding camera of Patent Literature 1 is connected via a wireand fixedly used, a relatively large-sized frame can be used therefor.In this case, it is possible to increase each angle of view of eachimaging device so as to overlap their imaging ranges with each other,thereby implementing the ranging function by a stereo method. However,in the case of mounting the ranging function and imaging function for afull solid angle on a miniaturized apparatus such as the portableinformation terminal, since the base length is shortened as a result inwhich the ranging function is implemented by the stereo method using theimaging device, there is a problem that the accuracy of ranging is notimproved. Therefore, applying the surrounding camera technique disclosedin Patent Literature 1 to a small apparatus without any modificationdoes not solve the problem.

The present invention has been made to solve the problems above, and anobject thereof is to provide a composite reception and emissionapparatus in which devices of a plurality of types are efficientlyarranged on a single base body.

Solution to Problem

In order to solve the problems above, the present invention includes thetechnical features described in the scope of claims. As one aspect ofthe present invention, it is provided a composite reception and emissionapparatus, comprising: a plurality of first-type devices that receivesor emits energy; a plurality of second-type devices that receives oremits energy, whose type is a different from a type of the first-typedevices; and a base body on which the plurality of first-type devicesand the plurality of second-type devices are mounted, when a receptiondirection or an emission direction of each of the plurality offirst-type devices is combined with each other for each of the pluralityof first-type devices, each of the plurality of first-type devicesreceiving the energy from an area of a full solid angle or emitting theenergy toward the area of the full solid angle, when a receptiondirection or an emission direction of each of the plurality ofsecond-type devices is combined with each other for each of theplurality of second-type devices, each of the plurality of second-typedevices receiving the energy from the area of the full solid angle oremitting the energy toward the area of the full solid angle, and theplurality of first-type devices and the plurality of second-type devicesbeing arranged on the base body so as to satisfy both of two constraintconditions below:

Constraint condition 1: A device closest to each of the plurality offirst-type devices in an orientation space is at least one of thesecond-type devices; and

Constraint condition 2: A device closest to each of the plurality ofsecond-type devices in the orientation space is at least one of thefirst-type devices.

Advantageous Effects of Invention

According to the present invention, it is possible to provide acomposite reception and emission apparatus in which devices of aplurality of types are efficiently arranged on a single base body. Theproblems, configurations, and effects other than those described abovewill be clarified by explanation of the embodiments below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an appearance of a composite reception and emissionapparatus 1 according to a first embodiment.

FIG. 2 is an overall block diagram of the composite reception andemission apparatus 1 according to the first embodiment.

FIG. 3A illustrates an arrangement example of first-type devices andsecond-type devices according to the first embodiment.

FIG. 3B illustrates an arrangement example of the first-type devices andthe second-type devices according to the first embodiment.

FIG. 4A illustrates an arrangement of devices by using an orientationgroup of face center (center of gravity) directions as viewed from thecenter of a cube.

FIG. 4B illustrates an arrangement of devices by using an orientationgroup of vertex directions.

FIG. 4C illustrates an arrangement of devices by using an orientationgroup of edge center orientations.

FIG. 5A illustrates an arrangement in which first-type devices arearranged on square-shaped faces of a cubic octahedron, and second-typedevices are arranged on equilateral triangle shaped faces thereof.

FIG. 5B illustrates each orientation to which each of all devices isdirected by using a polar coordinate system.

FIG. 6 illustrates the number of orientations and a required receptionand emission angle range for each orientation group included in aregular tetrahedron, a regular octahedron, and a cubic octahedron.

FIG. 7A illustrates an example in which a composite reception andemission apparatus is mounted on a portable information terminal havinga rectangular parallelepiped (hexahedron) shape.

FIG. 7B illustrates an arrangement in which each ranging sensor isarranged in each orientation of a vertex orientation group of a regularoctahedron.

FIG. 8 illustrates ranging data measured by the four ranging sensorsillustrated in FIG. 5A, respectively.

FIG. 9 illustrates full solid angle ranging data.

FIG. 10A illustrates an example of imaging data.

FIG. 10B illustrates an example of a ranging data synthesizing method.

FIG. 10C illustrates an example of connecting and shaping areas.

FIG. 11A illustrates an appearance of a composite reception and emissionapparatus according to a second embodiment.

FIG. 11B illustrates a display example of the composite reception andemission apparatus according to the second embodiment.

FIG. 12 illustrates entire blocks of the composite reception andemission apparatus according to the second embodiment.

FIG. 13 illustrates an algorithm of full solid angle imaging and rangingperformed by an image processor.

FIG. 14 illustrates an example in which distance information is addedand displayed in a full solid angle image.

FIG. 15 illustrates a flowchart of creating processing of a full solidangle image and full solid angle ranging data.

FIG. 16 illustrates a flowchart of reproduction processing of a fullsolid angle image and full solid angle ranging data.

FIG. 17A illustrates an example in which a composite reception andemission apparatus 1 is applied to an HMD.

FIG. 17B illustrates an example of a device arrangement on an HMD.

FIG. 18A illustrates a required reception and emission angle range ofeach device arranged on an HMD.

FIG. 18B illustrates each orientation to which each of all devices isdirected by using a polar coordinate system.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings. Throughout the drawings, the same componentsare provided with the same reference signs, and repetitive explanationtherefor will be omitted. In the following, as a combination of devicesof different types, an example of a combination of an imaging device(corresponding to a first-type device) and a ranging sensor(corresponding to a second-type device) will be described, meanwhile,the present invention is not limited thereto.

First Embodiment

In the first embodiment, in addition to a base body 2 on which devicesare mounted (see FIG. 1), an orientation defining polyhedron 5 (see FIG.3A) which defines the center orientation of each of the first-typedevice and the second-type device (orientation for directing the centerdirection of a reception and emission range). Here, the “orientation”means a direction in the real space to which the center direction of thereception and emission range is to be directed. Furthermore, an“orientation space” is a space defined as a set of orientations in thereal space. For example, the orientation in the real space is expressedby a deflection angle (θ, φ) of a polar coordinate system (r, θ, φ). Itis assumed that vectors in the real space in which one of them moves inparallel and overlaps on the other one of them have the sameorientations. The “base body 2” is an object that actually exists in thereal space, on which the first-type device and the second-type deviceare mounted. On the other hand, the “orientation defining polyhedron 5”is a conceptual form which is introduced to determine the centerorientation of each of the first-type device and the second-type devicemounted on the base body 2, and therefore, the “orientation definingpolyhedron 5” is not an actual object. The “orientation definingpolyhedron 5” defines, in the “orientation space”, an orientation groupincluding a plurality of orientations having good symmetries based onits geometric symmetrical property.

The orientation group is, for example, a group of orientations directedfrom the center position of the “orientation defining polyhedron 5”toward the centers of faces thereof having the same shapes (which willbe described later in detail). The “orientation defining polyhedron 5”is the shape having symmetries of at least two or more orientations of afirst symmetry and a second symmetry. Each of the first-type devicescorresponding to each of a plurality of orientations having the firstsymmetry included in the orientation defining polyhedron 5 is arrangedon the base body 2 with each corresponding orientation being set as thecenter orientation of each device. Furthermore, each of the second-typedevices corresponding to each of a plurality of orientations having thesecond symmetry included in the orientation defining polyhedron 5 isarranged on the base body 2 with each corresponding orientation beingset as the center orientation of each device. Both the arrangementpositions of the first-type devices and those of the second-type devicesare adjusted so that the base body 2 is not included in a reception andemission range of each device. Meanwhile, even if the base body 2obstructs the reception and emission range, the obstruction is allowablewhen the reception and emission range of each device is sufficientlywide and a combination of the reception and emission ranges of thedevices of the respective types covers a full solid angle.

Furthermore, if the obstructed orientation is an orientation which doesnot require any reception and emission, the obstruction in the receptionand emission range of a part of the devices relating to the orientationabove is also allowable. An arrangement with less obstruction by thebase body 2 is realized when the shape of the base body 2 is the same asthe shape of the orientation defining polyhedron 5 or when anarrangement orientation of each device as viewed from the center of thebase body 2 is close to the center orientation of each device. In thelatter arrangement, the center orientation direction of each device is,on average, close to the vertical direction of a surface of the basebody 2. Furthermore, in this arrangement, since the center orientationof each first-type device and that of each second-type device are set toform a nesting structure in the orientation space, the arrangementpositions thereof on the base body 2 also form the nesting structure,thereby improving the arrangement efficiency. In the case where theshape of the base body 2 is different from the shape of the orientationdefining polyhedron 5, positions in which the arrangement orientationsof the devices as viewed from the center of the base body 2 are thecenter orientations of devices are set as a basic arrangement. Then,considering the shape of the base body 2 and a limitation in mounting onthe base body 2, the arrangement positions on the base body 2 areadjusted by moving them from the basic arrangement while maintainingeach center position of each device. In this case, the arrangementefficiency of the devices is also good since the adjustment based on thebasic arrangement is performed.

FIG. 1 illustrates an appearance of a composite reception and emissionapparatus 1 according to the first embodiment. In the compositereception and emission apparatus 1, a body of a portable informationterminal, such as a smartphone, is the base body 2, and each of thefirst-type devices 1, 2, . . . , M (11-1, 11-2, . . . , 11-M) isarranged on the base body 2 with the orientation satisfying the firstsymmetry included in the orientation defining polyhedron 5 being set asthe center orientation. Furthermore, in the composite reception andemission apparatus 1, each of the second-type devices 1, 2, . . . , N(12-1, 12-2, . . . , 12-N) is arranged on the base body 2 with theorientation satisfying the second symmetry included in the orientationdefining polyhedron 5 being set as the center orientation. Anarrangement position of each device on the base body 2 is determined byconsidering a limitation in mounting on the base body 2. The orientationdefining polyhedron 5, the first symmetry, and the second symmetry willbe described later.

A controller 3 is provided inside the base body 2. The controller 3 isconnected to each of the first-type devices 1, 2, . . . , M (11-1, 11-2,. . . , 11-M) and each of the second-type devices 1, 2, . . . , N (12-1,12-2, . . . , 12-N). The controller 3 is configured by a computerincluding a processor and a circuit. Each of “M” indicating the numberof the first-type devices and “N” indicating the number of thesecond-type devices is an integer of two or more. “M” and “N” may be thesame number or different numbers. Here, the plurality of first-typedevices 1, 2, . . . , M (11-1, 11-2, . . . , 11-M) is collectivelyreferred to as a first-type device group 11, and the plurality ofsecond-type devices 1, 2, . . . , N (12-1, 12-2, . . . , 12-N) iscollectively referred to as a second-type device group 12.

Each of the first-type devices 1, 2, . . . , M (11-1, 11-2, . . . ,11-M) is an imaging device configured by using, for example, awide-angle lens and a Charge-Coupled Device (CCD) sensor or aComplementary metal-oxide-semiconductor (CMOS) sensor. Each of thefirst-type devices 1, 2, . . . , M (11-1, 11-2, . . . , 11-M) isarranged on an appropriate position of the base body 2 in an appropriateorientation, thereby enabling imaging of the full solid angle.

Each of the second-type devices 1, 2, . . . , N (12-1, 12-2, . . . ,12-N) is a TOF sensor for measuring a distance to, for example, a personand an object. Each of the second-type devices 1, 2, . . . , N (12-1,12-2, . . . , 12-N) is arranged on an appropriate position of the basebody 2 in an appropriate orientation, thereby enabling measurement of adistance to a full solid angle object.

FIG. 2 is an overall block diagram of the composite reception andemission apparatus 1 according to the first embodiment. The controller 3of the composite reception and emission apparatus 1 includes afirst-type device processor 211 configured to control the first-typedevice group 11, a second-type device processor 212 configured tocontrol the second-type device group 12, a Central Processing Unit (CPU)16, a Read Only Memory (ROM) 13 that holds programs and various settingvalues used to perform the processing of the CPU 16 and control of thefirst-type device group 11 and the second-type device group 12, a RAM 14that temporarily stores imaging data output from the first-type devicegroup 11 and ranging data output from the second-type device group 12(hereinafter, the imaging data and the ranging data are collectivelyreferred to as “measurement data”) and provides work areas for theprograms to be executed, an external memory 15 that stores themeasurement data, the imaging data, and video and image data generatedor captured in advance, and a system bus 17.

FIG. 3A illustrates an arrangement example of the first-type devices 1,2, . . . , M (11-1, 11-2, . . . , 11-M) and the second-type devices 1,2, . . . , N (12-1, 12-2, . . . , 12-N) based on the orientationdefining polyhedron 5, according to the first embodiment.

As the orientation defining polyhedron 5, a regular polyhedron, asemi-regular polyhedron, or a Catalan solid (a dual polyhedron of asemi-regular polyhedron) is used. In FIG. 3A, the shape of theorientation defining polyhedron 5 is a cubic octahedron.

Each of the first-type device group 11 and the second-type device group12 is arranged by using the symmetries included in the orientationdefining polyhedron 5. More specifically, by using the center of theorientation defining polyhedron 5 as a reference point, an orientationgroup including the orientations viewed from the center of theorientation defining polyhedron 5 toward the center of each face (whichmay be the center of gravity) is defined as the first symmetry, anorientation group including the orientations viewed from the center ofthe orientation defining polyhedron 5 toward the center of each edge isdefined as the second symmetry, and an orientation group including theorientations viewed from the center of the orientation definingpolyhedron 5 toward each vertex is defined as a third symmetry. Based onthe orientations of each orientation group included in the orientationdefining polyhedron 5 having good symmetries, which are set as thecenter orientations, the first-type devices 1, 2, . . . , M (11-1, 11-2,. . . , 11-M) and the second-type devices 1, 2, . . . , N (12-1, 12-2, .. . , 12-N) are arranged on the orientation defining polyhedron 5, andaccordingly, the full solid angle can be covered efficiently by anesting structure formed in the orientation space without interferenceof each device. For example, each of the first-type devices 1, 2, . . ., M (11-1, 11-2, . . . , 11-M) is arranged with a face centerorientation (first symmetry) being set as the center orientation whileeach of the second-type devices 1, 2, . . . , N (12-1, 12-2, . . . ,12-N) is arranged with a vertex center orientation (third symmetry)being set as the center orientation. Since devices of different typescan be arranged by the number of orientation groups, the number ofdevice types may be three or more.

Furthermore, in the case of the imaging device using a super wide anglelens such as a fisheye lens, since a single imaging device may have anangle of view of more than 180 degrees, it is not necessary to use allthe orientations in the orientation group to arrange devices. However,when such an imaging device is combined with different-type devices, theimaging device is arranged in a part of the orientation group that formsthe nesting structure with the orientation group in which the differenttype devices are arranged. As a result, it is possible to realize anefficient device arrangement with good symmetries as a whole whilereducing the interference therebetween. In this case, the nestingarrangement is formed in only the proximity of a small number of deviceswithin the orientation space. In this situation, it can be expressedthat in the nesting arrangement without deviation, considering mountingerrors, a first proximity device and a second proximity device of thesmaller number of devices are the different type devices.

The example illustrated in FIG. 3A is the arrangement example when thebase body 2 has the same shape as that of the orientation definingpolyhedron 5. In FIG. 3A, by using the orientation directed from thecenter of the cubic octahedron toward the square-shaped face centerwhich is set as the center orientation, the six first-type devices 11-1,11-2, 11-3 (FIG. 3A omits to illustrate the remaining three devices) arearranged on six positions, respectively. Each of the three remainingdevices that is not illustrated in FIG. 3A is arranged on asquare-shaped face positioned at the opposite invisible side.

Furthermore, by using the orientation directing from the center of thecubic octahedron toward the triangle-shaped face center which is set asthe center orientation, the eight second-type devices 12-1, 12-2, 12-3,12-4 (FIG. 3A omits to illustrate the remaining four devices) arearranged on eight positions, respectively. Each of the four remainingdevices that is not illustrated in FIG. 3A is arranged on an equilateraltriangle-shaped face positioned at the opposite invisible side.

An imagining range of each of the first-type devices 11-1, 11-2, 11-3and a ranging range of each the second-type devices 12-1, 12-2, 12-3,12-4 forms a reception and emission angle range of each device to coverthe full solid angle. When viewed from one of the first-type devices,the proximity device which is the closest thereto in the orientationspace is the second-type device, and when viewed from one of thesecond-type devices, the proximity device which is the closest theretoin the orientation space is the first-type device.

When each reception and emission angle range of the respective devicesis greater than a minimum required value, the orientation on which eachdevice is mounted needs not to exactly coincide with the symmetricalorientation included in the orientation defining polyhedron 5, and isprovided with high tolerance in its range covering the full solid angle.

As the first-type devices, for example, a plurality of imaging devicesto which a CCD or CMOS with a wide-angle lens is combined is arranged,which enables imaging of the full solid angle. As the ranging sensors asthe second-type devices, for example, a plurality of Time of Flight(TOF) sensors is arranged, which enables ranging of the full solidangle. The TOF sensor is used to measure a distance to a subject bymeasuring, for each pixel, a temporal deviation until the light emittedfrom the imaging element side is reflected by the subject. In this way,when the TOF sensor is used with a two-dimensional arrayed sensor (CMOSor CCD), the distance to the subject can be recognizedtwo-dimensionally.

In order to recognize the distance to the subject two-dimensionally, itis necessary to arrange the TOF sensor on the base body 2two-dimensionally. Even in the case of mounting a wide-angle lens, itsangle of view is about 120 to 140 degrees. On the other hand, since thesize of the pixel is about 10 μm, the size of the sensor itself can besuppressed to several millimeters square, thereby making it possible tomount a plurality of sensors on the base body 2. In this way, when thenumber of TOF sensors and their arrangement positions and angles areappropriately adjusted, it is easy to perform ranging of the full solidangle.

As long as the orientations to which the devices are directed are thesame as each other, the devices are not necessarily arranged on specificpositions (the face center position, the edge center position, and thevertex position) of the base body 2 (in this case, corresponding to theorientation defining polyhedron 5).

Furthermore, the shape of the base body 2 may be different from theshape of the orientation defining polyhedron 5. In this case, theorientation to which each device is directed is an orientation of theorientation group included in the orientation defining polyhedron 5. Forexample, in FIG. 3A, when a rectangular parallelepiped is employed asthe base body 2 without changing the orientations to which the imagingdevices and the ranging sensors are directed in the cubic octahedron,the imaging devices and the ranging sensors may be arranged on the facecenter positions and the vertex positions of the rectangularparallelepiped, respectively. FIG. 3B illustrates an arrangement exampleon the rectangular parallelepiped above.

In FIG. 3B, each of the imaging devices as the first-type devices isarranged on each face of the rectangular parallelepiped while each ofthe ranging sensors as the second-type devices is arranged on eachvertex of the rectangular parallelepiped. In this case, the orientationto which each of the imaging devices is directed is the same as theorientation directed from the center of the rectangular parallelepipedtoward the center of each face. On the other hand, the orientation towhich each of the ranging sensors is directed is different from theorientation directed from the center of the rectangular parallelepipedtoward each vertex, but the same as the orientation to which eachequilateral triangle-shaped face of the cubic octahedron in FIG. 3A isdirected. Since the shape of the base body 2 is the same as the shape ofa smartphone or tablet terminal (rectangular parallelepiped), theexample illustrated in FIG. 3B can be actually used as the portablecomposite reception and emission apparatus 1. Here, the above-described“orientation to which each of the imaging devices is directed” is theorientation of the center line of an angle of view of each imagingdevice, and the “orientation to which each of the ranging sensors isdirected” is the orientation of the center line of a scanning range ofeach ranging sensor.

FIG. 4A, FIG. 4B, and FIG. 4C illustrate an example in which each of thedevices described above is arranged by using each symmetry included in aregular polyhedron, a semi-regular polyhedron, and a Catalan solid,respectively. Each example in FIG. 4A, FIG. 4B, and FIG. 4C correspondsto the case where the shape of the base body 2 is the same as the shapeof the orientation defining polyhedron 5. FIG. 4A illustrates anarrangement of the devices by using the orientation group of the facecenter directions (center of gravity) as viewed from the center of thecube. FIG. 4B illustrates an arrangement of the devices by using theorientation group of the vertex directions. FIG. 4C illustrates anarrangement of the devices by using the orientation group of the edgecenter orientations.

Even on the same cubes, the number of devices to be mounted differsdepending on each orientation group. Which orientation group to be usedmay be determined based on a reception and emission angle of each deviceto be mounted. The details thereof will be described below.

With reference to FIG. 5 and FIG. 6, a reception and emission angle ofeach device, which is required to enable the ranging and photographingof the full solid angle to be performed by a combination of the devicesmounted thereon, will be described.

In FIG. 5A, each of the first-type devices (11-1), (11-2), (11-3) isarranged on a square-shaped face of a cubic octahedron (back side of thecubic octahedron is not illustrated), and each of the second-typedevices (12-1), (12-2), (12-3), (12-4) is arranged on an equilateraltriangle shaped face thereof (back side of the cubic octahedron is notillustrated). This example also corresponds to the case where the shapeof the base body 2 is the same as that of the orientation definingpolyhedron 5. In order to cover the full solid angle (to measure thefull solid angle) by the first-type devices mounted on the square-shapedfaces, the reception and emission angle range may cover an orientationrange within a pyramid formed by orientation vectors of a subgroupclosest thereto within the group of the first-type devices. That is, theorientation range (serving as a unit of a measurement range of thefirst-type device) of a triangular pyramid formed by X-axis, Y-axis, andZ-axis illustrated in FIG. 5A needs to be covered with margins (areaoverlapping with the reception and emission angle range of an adjacentfirst-type device) by the reception and emission angle range of thefirst-type device.

FIG. 5B illustrates each orientation to which each of all devices isdirected by using a deflection angle in a polar coordinate system. Sincethe first-type device (11-1) is directed to the orientation of θ=0degree, the circle illustrated in FIG. 5A is expressed by a strip shapein FIG. 5B. When the required reception and emission angle range isexpressed in FIG. 5B, the reception and emission angle range of one ofthe first-type devices (for example, the first-type device 11-2) maycover the central axis direction of the pyramid vertex angle, and the“central axis direction” is the direction of the second-type device(12-1) in FIG. 5A. That is, the “central axis direction” is theV-direction illustrated in FIG. 5A, and is the value of α illustrated inFIG. 5B. That is, the required reception and emission angle range istwice the angle α formed by the central axis direction of the pyramidvertex angle and the direction of the first-type device.

In FIG. 5B, the required reception and emission angle range is indicatedby a broken line around the first-type device 11-2. In this case, sincethe angle α is 54.7°, the required reception and emission angle is109.5° which is twice the angle α. Based on this example, FIG. 6illustrates the number of orientations (may be the number of devices tobe mounted) and a required reception and emission angle range for eachorientation group included in a regular tetrahedron, cube, regularoctahedron, and cubic octahedron. Here, as illustrated in the case ofthe face center orientation group of the cubic octahedron, theorientation group may be further divided in details depending on thetypes of the face, edge, and vertex. The above is merely an example, andin the case of other polyhedrons, the required reception and emissionangle range can be obtained by the same approach. In the above, therequired reception and emission angle range of each of the first-typedevices has been described, meanwhile, the required reception andemission angle range of each of the second-type devices can bedetermined by the same approach.

In the case of the above-described required reception and emissionangle, when TOF sensors are used as the ranging sensors, measurementranges of the adjacent TOF sensors overlap with each other therebetween.With this regard, when the data of each sensor is complemented with eachother, it is possible to achieve the measurement with high accuracy.

That is, in the case of a TOF sensor, since the measurement accuracy ofits data closer to the center of the angle of view is higher than thatof the peripheral portion of the angle of view. Accordingly, bycomparing each piece of data of the adjacent TOF sensors and using theranging data which is closer to the center portion of the angle of view,it is possible to achieve the ranging of the full solid angle with highaccuracy. Each required reception and emission angle illustrated in FIG.6 indicates an angle of view required for the TOF sensor when arrangingeach TOF sensor in each orientation group of the polyhedron. Forexample, in the case of arranging each TOF sensor on each vertex of aregular hexahedron, the number of devices to be arranged is eight, andthe angle of view of each TOF sensor required to perform ranging of thefull solid angle is 109.5 degrees. Similarly, in the case of arrangingeach TOF sensor on each vertex of a cubic octahedron, the number ofsensors to be arranged is twelve, and the required angle of view thereofis 90 degrees. Accordingly, in this case, even TOF sensors with smallangles of view can perform ranging of the full solid angle. In the casewhere the angles of view overlap with each other between the adjacentsame type of devices, as described above, measurement data and imageswith high accuracy can be obtained by complementing the measurement dataof the adjacent devices. In the above, the example of the TOF sensors asthe ranging sensors has been described. Meanwhile, similarly, whenpieces of the imaging data of adjacent imaging devices overlap with eachother, since an image closer to the center of the angle of view has lessdistortion, it is preferable to select the image closer to the center ofthe angle of view. As described above, the reception and emission anglerange of a device is an important factor in determining the number ofdevices to be mounted.

FIG. 6 illustrates a table in which the orientation group, the number oforientations included therein, and required reception and emission angleranges are summarized for each various polyhedron. Although FIG. 6illustrates the cases of a regular tetrahedron, cube, regularoctahedron, and cubic octahedron as examples of polyhedrons, the presentinvention is not limited thereto. The required reception and emissionangle ranges for various polyhedrons such as other regular polyhedrons,semi-regular polyhedrons, and Catalan solids can be determined by theapproach described above, and the number of devices to be mountedthereon can also be determined by this approach.

That is, the present embodiment relates to a combination of full solidangle ranging performed by the ranging sensors, each of whichmeasurement range is a small full solid angle, and full solid angleimaging performed by the cameras. In order to efficiently cover the fullsolid angle, in the present embodiment, a combination of orientationshaving good symmetries are selected for both the ranging system and theimaging system. With this regard, the symmetries included in a regularpolyhedron, a semi-regular polyhedron, and a Catalan solid which serveas the orientation defining polyhedron 5 are used. Specifically, basedon the orientation groups of the face center orientations (center ofgravity), edge center orientations, and vertex orientations in each ofthe regular polyhedron, a semi-regular polyhedron, and a Catalan solidas viewed from each center thereof, the ranging sensors and the imagingdevices are arranged in the orientations of the orientation groupsincluded in each of the various regular polyhedrons (or semi-regularpolyhedrons or Catalan polyhedrons). As a result, it is possible toefficiently cover the full solid angle by the nesting form withoutinterference therebetween.

In this connection, the imaging devices each having a wide-angle lenswhose angle of view is more than 180 degrees enable imaging of the fullsolid angle when each of them is arranged in two opposite directions,respectively. In this case, it is not necessary to arrange the devicesin all the orientations of the orientation group. However, also in thiscase, when the different type devices are combined thereto, each of theimaging devices is arranged on a part of the orientation group whichforms the nesting structure with the orientation group in which each ofthe different type devices is arranged. As a result, it is possible toachieve an efficient device arrangement with good symmetries as a wholewhile reducing interference between the devices of different types.

As will be described later as a modification, the axis passing throughthe center of each device does not need to be gathered in one point aslong as the orientation thereof is not changed. According to thismodification, the degree of flexibility in arrangement is increased.Furthermore, as also will be described as another modification, in thecase where the reception and emission angles for covering differ in thevertical and horizontal directions of the device, the arrangementefficiency may be improved when the face center orientations of arhombic polyhedron are used.

Next, with regard to the above-described arrangement positions of theimaging devices and ranging sensors, an example of a specificarrangement method will be described with reference to FIG. 7A and FIG.7B. The following explanation is an example, and thus the arrangementexample is not limited thereto.

FIG. 7A illustrates an example in which the composite reception andemission apparatus 1 is mounted on a portable information terminalhaving an approximately rectangular parallelepiped (hexahedron) shape.The approximate rectangular parallelepiped is the base body 2. Forexample, by setting a regular octahedron as the orientation definingpolyhedron 5 and using the orientation group thereof, the rangingsensors and the imaging devices are arranged thereon. When theorientation group of the face center orientations is used as the centerorientation of each ranging sensor, it is necessary to arrange eightsets of the ranging sensors. However, since the portable informationterminal is a hexahedron, it is impossible to mount each one of theeight ranging sensors on each face of the hexahedron. To solve thisproblem, for example as illustrated in FIG. 7A, four sets of the eightsensors are arranged on the edge portions of the top face and other foursets of the eight sensors are arranged on the edge portions of thebottom face while being directed to the orientations of the face centerorientation group of the regular octahedron.

When each ranging sensor is arranged on each edge of the portableinformation terminal, it may be mounted on a groove formed (or providedby chamfering) on each edge. In this case, it is sufficient that theangle of view of each ranging sensor is equal to or more than 109.5degrees which are the value for the face center orientations of theorientation group of the regular octahedron illustrated in FIG. 6. Inthe above, the example of the method of arranging each ranging sensor byforming a groove on each edge of the portable information terminal hasbeen described. Meanwhile, the arrangement method is not limited theretoas long as the center orientation of each ranging sensor is directed inthe orientation described above.

On the other hand, the imaging devices 110, 111 are arranged in thedirections of the edge center orientation group of the regularoctahedron. Here, it is assumed that the angle of view of each imagingdevice is equal to or more than 180 degrees. In this case, twoorientations facing each other are selected from the edge centerorientation group, and each device is mounted on each of the two faceson the symmetrical positions of the hexahedron of the portableinformation terminal. As a result, it is possible to perform imaging ofthe full solid angle.

FIG. 7B illustrates an example of arrangement of each ranging sensor, inwhich each orientation of the vertex orientation group of a regularoctahedron serving as the orientation defining polyhedrons 5 is used asthe center orientation. In this case, three sets of the ranging sensorsare arranged on the top face and other three sets of the ranging sensorsare arranged on the bottom face. More specifically, each vertex of theportable information terminal is cut off obliquely, and each rangingsensor is mounted thereon to be directed in the orientation of theabove-mentioned orientation group. Also in this case, it is sufficientthat the angle of view of each ranging sensor is equal to or more than109.5 degrees.

On the other hand, the imaging devices 110, 111 are arranged by usingthe orientations of the face center orientation group of a regularoctahedron as the center orientations. Here, it is assumed that theangle of view of each imaging device is equal to or more than 180degrees. In this case, two orientations facing each other are selectedfrom the face center orientation group, and each device is mounted oneach of the two faces on the symmetrical positions of the hexahedron ofthe portable information terminal. As a result, it is possible toperform imaging of the full solid angle. In this example, although allorientations of the face center orientation group are not provided withthe imaging sensors, the device closest to each of the all rangingsensors in the orientation space is an imaging device.

As described above, since the orientations of the orientation groupincluded in a regular polyhedron, a semi-regular polyhedron, and aCatalan solid which serve as the orientation defining polyhedron 5 areused as the center orientations, it is possible to efficiently arrangethe imaging devices and the ranging sensors on the base body 2, andperform ranging and imaging of the full solid angle.

In this connection, considering a case of taking images at night or in adark place, there may be a demand of an illumination device for a fullsolid angle. In this regard, a plurality of illumination devices may bemounted by the same approach as described above. The devices arearranged such that the center orientation of each imaging device and thecenter orientation of each illumination device form a nesting structurein the orientation space, thereby making it possible to prevent theimaging devices from being directly interfered by the illuminationdevices.

FIG. 8 illustrates an example in which multiple pieces of data measuredrespectively by the four ranging sensors 12-1, 12-2, 12-3, 12-4illustrated in FIG. 5A are synthesized to generate one piece of rangingdata.

Each ranging sensor 12-1, 12-2, 12-3, 12-4 includes its measurementrange. In each measurement range, a ranging error becomes large in aperipheral portion. This is because, for example, since a TOF sensoruses an optical system lens as its measurement system, distortion in theperipheral portion increases in the same manner as an angle of view of acamera sensor. Accordingly, as illustrated in FIG. 8, the measurementaccuracy in the peripheral portion is lower than that in the centralportion.

For the reason above, it is not preferable to use the peripheralportion. Accordingly, each of the ranging sensors 12-1, 12-2, 12-3, 12-4is arranged such that the measurement ranges of the adjacent rangingsensors overlap with each other in the orientation space so as to allowthe portions with high measurement accuracy to be used preferentially.As a result, it is possible to realize ranging of the full solid anglewith high accuracy. Since the positional relationships between theadjacent ranging sensors are known in advance, for example, at the timeof layout thereof, a ranging territory of each ranging sensor may bepredetermined for the selection above. The same procedure is applied tothe remaining ranging sensors 12-5 to 12-8 illustrated in FIG. 7A toimprove the accuracy of ranging of the full solid angle.

FIG. 9 illustrates an example of synthesizing multiple pieces of rangingdata into one piece of ranging data. Although FIG. 9 illustrates thedistance information by using three types of patterns, the presentinvention is not limited thereto. In practice, the distance iscalculated more accurately with shading, or may be color-coded.

The CPU 16 may read the ranging data from each of the ranging sensors12-1 to 12-4 and synthesizes and trims them to generate full solid angleranging data 800. The ranging data synthesizing processing may beperformed using only ranging data of each ranging sensor, or may beperformed by complementing the ranging data using the imaging dataobtained from the imaging devices. The details thereof will be describedlater.

At the time of synthesizing the ranging data, there is a possibilitythat deviation in the ranging data is generated at a connection portionof the measurement ranges. The deviation is generated because geometricfocal points of lights incident on each ranging sensor does notnecessarily coincide therewith, and thus an area of the orientationrange covered by each ranging sensor is deviated depending on thedistance to the measurement object. A data correction method in thiscase will be described with reference to FIG. 10.

The imaging data illustrated in FIG. 10A is based on the example of theimaging device 11-1 of FIG. 2. FIG. 10B illustrates a method ofsynthesizing two pieces of data of the ranging sensors 12-1, 12-2 thatperform ranging of the range of the above-described imaging data. In theactual case, since four sets of the ranging sensors are arranged asproximity devices of the imaging device 11-1, four pieces of data aresynthesized. However, in order to simplify the explanation, an exampleof synthesizing two pieces of data of two proximity devices will bedescribed. Furthermore, in the actual case, portions other than theimaging range of the imaging device 11-1 exist in the measurement rangeof each of the two ranging sensors 12-1, 12-2. However, for the purposeof simplifying the explanation, measurement within the imaging range ofthe imaging device 11-1 will be described in the following. When it isassumed that the ranging sensors 12-1, 12-2 are arranged atgeometrically accurate positions, angles, and directions, only thedistortion in the peripheral portion needs correction. In this case, theranging data may be synthesized by using the data with high accuracy inview of the measurement ranges and their positional relationship asillustrated in FIG. 8.

On the other hand, the ranging sensors are arranged on an actual productin a state of being deviated from the geometrically accurate positions.That is, there are cases where “focal points” (a point on which incidentsignals are geometrically converged) of each device are not aligned atone point, or pieces of data are not neatly connected with each otherdue to problems of distortion, mounting positions of the rangingsensors, or accuracy even when the angles of view overlap with eachother. Accordingly, it is necessary to dynamically adjust the connectionof data in accordance with the distance to the subject by using theimaging data. For example, as illustrated in FIG. 10B, a ranging datarange obtained by measuring a certain target by means of the rangingsensors 12-1, 12-2 is slightly deviated due to deviation in the focalpoints and mounting accuracy. Accordingly, the ranging data is correctedto bring the deviation to match the above-described imaging data, andshaping of the connection between the areas (see FIG. 10C) is performed.The correction can be calculated based on the distance to the object andthe property and arrangement of the device, meanwhile, when the shapingis further performed with reference to the imaging data, it is possibleto improve the accuracy of the data in the boundary area.

According to the first embodiment, in the case of arranging a pluralityof devices of multiple types on one base body 2, orientation groupshaving different symmetries included in the orientation definingpolyhedron 5 are used so that the devices are arranged to form nestingstructures in the orientation space. As a result, it is possible toefficiently arrange the devices of both the same type and differenttypes on the base body 2 without interference between their centerorientations, and realize the ranging of the full solid angle.

Second Embodiment

A second embodiment is an embodiment in which a composite reception andemission apparatus 1 a is applied to a portable information terminal 4.FIG. 11A illustrates an appearance of the composite reception andemission apparatus 1 a according to the second embodiment. FIG. 11Billustrates a display example of the composite reception and emissionapparatus 1 a.

The composite reception and emission apparatus 1 a includes, on theportable information terminal 4 as the base body 2, a display 20 forconfirming a distance and image of an object or a person measured by theimaging devices and the ranging sensors. The configurations of theimaging devices and the ranging sensors are the same as thoseillustrated in FIG. 5B. FIG. 11A does not illustrate invisible portions.

As illustrated in FIG. 11B, the composite reception and emissionapparatus 1 a may display a character or the like to show thatmeasurement is being performed. After ranging and imaging, the compositereception and emission apparatus 1 a may display the captured image ofthe full solid angle and the distance information. By rotating theportable information terminal 4 in, for example, the up and down andleft and right directions, a full solid angle image is displayed on thedisplay 20. At this time, the composite reception and emission apparatus1 a cooperates with a gyro sensor of the portable information terminal(see FIG. 12) to confirm the orientation of the portable informationterminal 4, and displays an image corresponding thereto.

FIG. 12 illustrates entire blocks of the composite reception andemission apparatus 1 a.

The portable information terminal 4 includes a camera 110 having awide-angle lens and an imaging element (for example, a CCD sensor or aCMOS sensor), which serves as the first-type device, an in-camera 111having a wide-angle lens and an imaging element as well, whose imagingrange covers the display 20 side, an image processor 112 configured toperform image processing based on the image data obtained from each ofthe camera 110 and the in-camera 111, the N number of TOF sensors 1, 2,. . . , N (12-1), (12-2), . . . , 12-N) which serve as the second-typedevices, a ROM 13, a RAM 14, an external memory 15, a CPU 16, a systembus 17, a gyro sensor 18, an acceleration sensor 19, the display 20, aLAN communication unit 21, a telephone network communication unit 22,and a Global Positioning System (GPS) 23.

The image processor 112 includes a distortion correction section 114configured to correct an image that is distorted from the captured imagedue to angle correction, rotation correction, or a wide-angle lens sothat it appears to be the original state, a subject recognition section116 configured to recognize the face of a person or an object, atrimming section 115 configured to cuts out a portion of the face of theperson or the object from the image data base on the subject recognitionsection 116, and an image synthesis section 117 configured to synthesizethe images captured by the plurality of cameras.

The second embodiment includes at least two cameras. In theabove-described example, two cameras such as the in-camera and thenormal camera each having a wide-angle lens whose angle of view is atleast 180 degrees or more are provided, and each of the cameras ismounted on each different face, thereby making it possible to take animage of the full solid angle. The number of cameras to be mounted isnot limited thereto as long as being two or more.

In the image processor 112, firstly, the distortion correction section114 performs distortion correction on the image data acquired fromeither the camera 110 or the in-camera 111, and after the subjectrecognition section 116 recognizes a person or an object, the trimmingsection 115 trims the person or the object. Meanwhile, the distortioncorrection is not limited thereto, and it may be configured to performdistortion correction with high accuracy after simply correcting thedistortion generated due to a super wide angle lens, or display, on thedisplay 20, a detailed distance to the person or the object obtainedfrom the ranging sensor group 12, which will be described later.

FIG. 13 illustrates an algorithm of the full solid angle imaging andranging performed by the image processor 112.

The image processor 112 includes a first distortion correction section114-1 configured to receive the imaging data from the camera 110 andcorrect the distortion due to the wide-angle lens, and a seconddistortion correction section 114-2 configured to receive the imagingdata from the in-camera 111 and correct the distortion due to thewide-angle lens. The first distortion correction section 114-1 includesa first aberration correction section and a first normal imageconversion section. Similarly, the second distortion correction section114-2 includes a second aberration correction section and a secondnormal image conversion section. In the example above, one distortionprocessing unit is provided for one camera. Meanwhile, the presentinvention is not limited thereto, and one processing function mayperform the processing while switching if it can perform high-speedprocessing and there is no time constraint.

Each of the first distortion correction section 114-1 and the seconddistortion correction section 114-2 outputs, to the image synthesissection 117, the image after the distortion correction processing. Theimage synthesis section 117 synthesizes the image after the distortioncorrection acquired from the first distortion correction section 114-1and the image after the distortion correction acquired from the seconddistortion correction section 114-2 so as to create an image of the fullsolid angle (hereinafter referred to as “full solid angle image”), andoutputs the created image to the subject recognition section 116.

The subject recognition section 116 performs subject recognitionprocessing based on the full solid angle image, and outputs a resultthereof to the trimming section 115.

The trimming section 115 trims, in the full solid angle image, the imagearea of the subject recognized in the subject recognition processing.The trimming section 115 may completely trim the subject area to extractonly the subject area, or perform processing of adding a frame thatencloses a frame of the subject area. The trimmed image may be furthersubjected to the distortion correction, or a distance thereto may bedisplayed by enlarging or emphasizing the trimmed portion, which will bedescribed later.

On the other hand, the CPU 16 acquires the ranging data of each of theTOF sensors 12-1, 12-2, . . . , 12-n and synthesizes them into the fullsolid angle ranging data. The CPU 16 complements a plurality of piecesof ranging data measured by the angle of view of the ranging sensorswith each other to create the full solid angle ranging data 800 (seeFIG. 8 and FIG. 9).

The CPU 16 acquires the full solid angle image from the image synthesissection 117 (L-A in FIG. 13), and makes it associated with the fullsolid angle ranging data 800. There are various kinds of methods forthis processing. For example, a subject is placed in advance on a knownposition around the composite reception and emission apparatus 1 a tocreate the full solid angle image, while a distance from the compositereception and emission apparatus 1 a to the subject above is measured byusing the ranging sensor group 12 to create the full solid angle rangingdata 800. Then, the subject area in the full solid angle image in whichthe subject above is captured is made associated with the distanceinformation to the subject above read from the full solid angle rangingdata 800 to create calibration data in which the imaging area in thefull solid angle image and the distance information to the subjectcaptured in the imaging area are associated with each other.

When the user uses the composite reception and emission apparatus 1 a,the CPU 16 creates the full solid angle image, refers to the calibrationdata, and reads the distance information corresponding to the imagingarea in the full solid angle image. The CPU 16 outputs the read distanceinformation and the corresponding position of the imaging area in thefull solid angle image (for example, coordinates in the full solid angleimage) to a distance information addition section 118.

The distance information addition section 118 adds the distanceinformation acquired from the CPU 16 to the subject area in the fullsolid angle image based on the information of the subject area acquiredfrom the trimming section 115.

FIG. 14 illustrates an example in which the distance information isadded and displayed in the full solid angle image. In FIG. 14, a displayrange of the display 20 displays a part of the full solid angle. Thefull solid angle may be displayed by rotating the position of thedisplay 20 in the up and down and left and right directions based on thedata of the gyro sensor 18, or scrolling the display 20 in the back andforth and left and right directions. Furthermore, as illustrated in FIG.14, the distance information may be displayed for a specific object.

FIG. 15 illustrates a flowchart of processing of creating the full solidangle image and the full solid angle ranging data 800 in the compositereception and emission apparatus 1 a. The flow of the processingdescribed below is the same as that of the composite reception andemission apparatus 1 described in the first embodiment.

The user of the composite reception and emission apparatus 1 a whowishes to perform imaging and ranging of the full solid angle switchesan operation mode of the composite reception and emission apparatus 1 ato a full solid angle imaging and ranging mode (step S100). Thecomposite reception and emission apparatus 1 a displays a mode selectionscreen on the display 20 to receive a mode selection operation from theuser. The composite reception and emission apparatus 1 a may beconfigured to, when the operation mode is switched to the full solidangle imaging and ranging mode, cause the display 20 to display, forexample, “full solid angle imaging and ranging system is underoperation” in order to urge the user to pay attention, for example, notto shake the apparatus as much as possible.

Each of the camera 110 and the in-camera 111 is activated to startcapturing an image. Furthermore, each of the ranging sensors 12-1, 12-2,. . . , 12-N is activated to start ranging (step S101).

The image data generated by each of the camera 110 and the in-camera 111is output to the image processor 112. The image processor 112 correctsthe distortion caused by the wide-angle lens (step S102), and creates afull solid angle image (step S103).

The CPU 16 (second-type device processor 212 in the first embodiment)synthesizes the ranging data measured by the ranging sensors 12-1, 12-2,. . . , 12-N to create the full solid angle ranging data 800 (stepS104). Step S103 and step S104 may be executed simultaneously, or stepS104 may be executed firstly.

The image processor 112 detects a subject area from the full solid angleimage (step S105), and performs the trimming processing thereon to cutout the subject area. Then, the image processor 112 adds the distanceinformation from the composite reception and emission apparatus 1 a tothe subject by using the full solid angle ranging data 800 created instep S104 (S106).

The image processor 112 records the full solid angle image to which thedistance information has been added and the full solid angle rangingdata in at least one of the RAM 14 and the external memory 15 (stepS107), and ends the processing.

FIG. 16 illustrates a flowchart of reproduction processing of the fullsolid angle image and the full solid angle ranging data 800 in thecomposite reception and emission apparatus 1 a. The flow of theprocessing described below is the same as that of the compositereception and emission apparatus 1 described in the first embodiment.

The composite reception and emission apparatus 1 a performs an operationfor shifting to a reproduction mode of the full solid angle image andthe full solid angle ranging data (step S110). The composite receptionand emission apparatus 1 a displays a mode selection screen on thedisplay 20 to receive a mode selection operation from the user.Furthermore, the composite reception and emission apparatus 1 a receivesan operation for selecting a file to be reproduced out of the datarecorded in the RAM 14 or the external memory 15 from the user, andreproduces the file (step S111).

The file to be reproduced includes the image information and thedistance information of a range wider than the display 20, that is, theimage information and the distance information of the full solid angle(see FIG. 14). Accordingly, the composite reception and emissionapparatus 1 a calculates the orientation information and the elevationinformation in the horizontal plane based on the position informationfrom the gyro sensor 18, the acceleration sensor 19, and the GPS 23 todisplay the image of a particular orientation in the full solid angleimage (partial image of the full solid angle image) and the distanceinformation.

Thereafter, the composite reception and emission apparatus 1 a requeststhe user to scroll the display 20 or move the composite reception andemission apparatus 1 a (step S112).

When detecting the movement of the user (for example, scrolling ormoving the composite reception and emission apparatus 1 a) (stepS113/Yes), the composite reception and emission apparatus 1 a displaysan image and the distance information of the instructed direction (stepS114).

When not detecting the movement of the user (step S113/No) nor atermination condition in which, for example, any instruction or movementof the user is absent for a predetermined period after step S114, orwhen receiving an operation for terminating the reproduction (stepS115/Yes), the composite reception and emission apparatus 1 a terminatesthe reproduction processing. When the termination condition is notsatisfied (step S115/No), the processing returns to step S112.

FIG. 17A illustrates an example in which the composite reception andemission apparatus 1 is applied to a Head Mounted Display (HMD) 50.

At the time of displaying Augmented Reality (AR) by using the HMD,imaging data and ranging data obtained by performing imaging and rangingin the real surrounding environment of an HMD user may be used. As aresult, it is possible to improve the control accuracy in superimposingand displaying a virtual object on the real space.

As illustrated in FIG. 17A, an HMD 50 includes a side head attachment50-1, a top head attachment 50-2, and a transmission or non-transmissiontype display 50-3. The HMD 50 is provided with a first camera 51-1having a wide-angle lens on the center of the upper side of the display50-3, and a second camera 51-2 having a wide-angle lens on the center ofthe rear head part of the side head attachment 50-1. The HMD 50 is alsoprovided with a first ranging sensor 52-1 and a second ranging sensor52-2 on each of the left and right vertexes of the display 50-3,respectively. In addition, the HMD 50 is provided with a third rangingsensor 52-3 on the top head prat of the top head attachment 50-2, and afourth ranging sensor 52-4 on the center of the rear head part of theside head attachment 50-1. That is, as illustrated in FIG. 17B, thepresent example of the HMD 50 uses a regular tetrahedron as theorientation defining polyhedron 5. Each of the cameras has an angle ofview of 180 degrees or more, and is arranged on two opposingorientations (51-1, 51-2), respectively, of the edge center orientationgroup while each of the ranging sensors is arranged in each of thevertex center orientations.

In the case of creating a full solid angle image and full solid angleranging data in the HMD 50, it is desirable to place the connectionportion in the full solid angle image directly below the HMD 50, inother words, on the side of the wearer of the HMD 50. This is becausethe wearer of the HMD 50 mostly does not need his or her own image. Thisis also the case of the ranging data. Since the wearer of the HMD 50rarely needs to know the distance to himself or herself, it is desirableto place the connection portion in the ranging data on the side of thewearer of the HMD 50. In this connection, FIG. 18A and FIG. 18Billustrate a modification of a sensor arrangement. In this modification,since an orientation of each camera is deviated from the edge centerorientation while ensuring the required reception and emission anglerange with an adjacent sensor, an orientation of each sensor is adjustedso that each connection portion in the imaging data and the ranging datais placed directly below the HMD 50. FIG. 18B illustrates orientationsto which devices are directed in the orientation space. In FIG. 18B,orientations of devices 51-1, 51-2 arranged in the edge centerorientations are adjusted. When the positive direction of the Z-axis isselected in the vertically upper direction, the user's body is in thenegative direction of the Z-axis and the optical axis of the camera isin the horizontal direction. As a result, each connection portion inboth the imaging data and the ranging data is placed on the side of thewearer of the HMD 50. In this modification as well, devices closest toeach device in the orientation space are the different type devices.

According to the present embodiment, it is possible to arrange devicesof different types on a small portable information terminal such as asmart phone such that the arrangement positions of the devices of bothdifferent types and the same type do not overlap with each other, aswell as the measurement ranges thereof include the full solid angle.

The embodiment described above is a merely example of an embodiment ofthe present invention, and the present invention is not limited thereto.For example, as one of the modifications, the present invention may beused for a video conference system. When distance information and objectrecognition of a full solid angle is used in the video conferencesystem, a position of each person attending a conference can berecognized based on face recognition images and the distance informationof all attendees. As a result, it is possible to specify a person who isspeaking, and enlarge and display the speaking person by a combinationwith a directional microphone, or specify the content of conversationand the attendees, and record the conversation clearly.

A transmission antenna and a reception antenna of radio waves areincluded in the examples of the reception device and the emissiondevice. Since the frequency used in a cell phone becomes high and thedirectivity thereof becomes strong, it is necessary to efficientlyarrange antennas so as to cover a full solid angle as reception andemission ranges of the radio waves. The present invention is alsoeffective in this case.

Furthermore, a microphone and a speaker, each having a strongdirectivity, are included in the examples of the reception device andthe emission device. The present invention is also effective in an audioinput and output device such as a smart speaker so as to realize soundexchange for a specific user targeted from among users existing nearby.

Each of the hardware configuration of the composite reception andemission apparatus 1 and that of the composite reception and emissionapparatus 1 a is merely an example. A single set of the CPU 16 mayperform each function of the first-type device processor 211, thesecond-type device processor 212, and the image processor 112.

REFERENCE SIGNS LIST

-   1, 1 a: composite reception and emission apparatus-   2: base body-   3: controller-   4: portable information terminal-   5: orientation defining polyhedron-   11: first-type device group-   12: second-type device group-   13: ROM-   14: RAM-   15: external memory-   16: CPU-   17: system bus-   18: gyro sensor-   19: acceleration sensor-   20: display-   21: LAN communication unit-   22: telephone network communication unit-   23: GPS-   50: HMD-   50-1: side head attachment-   50-2: top head attachment-   50-3: display-   51-1: first camera-   51-2: second camera-   52-1: first ranging sensor-   52-2: second ranging sensor-   52-3: third ranging sensor-   52-4: fourth ranging sensor-   110: camera-   111: in-camera-   112: image processor-   114: distortion correction section-   114-1: correction section-   114-2: correction section-   115: trimming section-   116: subject recognition section-   117: image synthesis section-   118: distance information addition section-   211: first-type device processor-   212: second-type device processor-   800: full solid angle ranging data

1. A composite reception and emission apparatus, comprising: a plurality of first-type devices that receives or emits energy; a plurality of second-type devices that receives or emits energy, whose type is a different from a type of the first-type devices; and a base body on which the plurality of first-type devices and the plurality of second-type devices are mounted, when a reception direction or an emission direction of each of the plurality of first-type devices is combined with each other for each of the plurality of first-type devices, each of the plurality of first-type devices receiving the energy from an area of a full solid angle or emitting the energy toward the area of the full solid angle, when a reception direction or an emission direction of each of the plurality of second-type devices is combined with each other for each of the plurality of second-type devices, each of the plurality of second-type devices receiving the energy from the area of the full solid angle or emitting the energy toward the area of the full solid angle, and the plurality of first-type devices and the plurality of second-type devices being arranged on the base body so as to satisfy both of two constraint conditions below: Constraint condition 1: A device closest to each of the plurality of first-type devices in an orientation space is at least one of the second-type devices; and Constraint condition 2: A device closest to each of the plurality of second-type devices in the orientation space is at least one of the first-type devices.
 2. A composite reception and emission apparatus, comprising: at least three first-type devices that receive or emit energy; at least two second-type devices that receive or emit energy, whose type is a different from a type of the first-type devices and which are less in number than the first-type devices; and a base body on which the first-type devices and the second-type devices are mounted, when all the energy received by each of the first-type devices are combined, the first-type devices receiving the energy from an area of a full solid angle, or when all the energy emitted from each of the first-type devices are combined, the first-type devices emitting the energy toward the area of the full solid angle, and when all the energy received by each of the second-type devices are combined, the second-type devices receiving the energy from the area of the full solid angle, or when all the energy emitted from each of the second-type devices are combined, the second-type devices emitting the energy toward the area of the full solid angle.
 3. The composite reception and emission apparatus according to claim 1, wherein an orientation defining polyhedron is defined, the orientation defining polyhedron has a shape including a first symmetry with one point in the orientation defining polyhedron as a reference point, and a second symmetry different from the first symmetry, each of the plurality of first-type devices is arranged in an orientation satisfying the first symmetry included in the orientation defining polyhedron, and each of the plurality of second-type devices is arranged in an orientation satisfying the second symmetry included in the orientation defining polyhedron.
 4. The composite reception and emission apparatus according to claim 3, wherein the orientation defining polyhedron is a regular polyhedron, a semi-regular polyhedron, or a Catalan solid.
 5. The composite reception and emission apparatus according to claim 3, wherein each of the first symmetry and the second symmetry is one of an orientation group directed from the reference point toward a face center or a center of gravity of the orientation defining polyhedron, an orientation group directed from the reference point toward an edge center orientation of the base body, or an orientation group directed from the reference point toward a vertex orientation of the base body.
 6. The composite reception and emission apparatus according to claim 1, wherein each of the plurality of first-type devices is a reception device and each of the plurality of second-type devices is an emission device, or each of the plurality of first-type devices is the emission device and each of the plurality of second-type devices is the reception device, and each of the plurality of first-type devices and the plurality of second-type devices is arranged on the base body in an orientation and a position on which the reception device does not directly receive the energy emitted from the emission device.
 7. The composite reception and emission apparatus according to claim 1, wherein the base body is a portable information terminal.
 8. The composite reception and emission apparatus according to claim 1, further comprising a processor configured to synthesize information measured by the plurality of first-type devices to generate information of the full solid angle, and synthesize information measured by the plurality of second-type devices to generate information of the full solid angle.
 9. The composite reception and emission apparatus according to claim 1, wherein either the first-type devices or the second-type devices are imaging devices, and the others are ranging sensors.
 10. The composite reception and emission apparatus according to claim 9, further comprising a processor configured to perform synthesize by using imaging data measured by the imaging devices and ranging data measured by the ranging sensors so as to display image information of the full solid angle.
 11. The composite reception and emission apparatus according to claim 10, wherein the processor is further configured to, when synthesizing the ranging data as ranging information of the full solid angle, extract, for each measurement range of the ranging sensors, the ranging data obtained from orientation angles measured with high accuracy, and synthesize the ranging data as extracted. 